Exploring novel quantum phases and the associated phase transitions in systems with strong electron correlations is a major subject of contemporary condensed matter physics.^[1–3] In this context, heavy fermion (HF) compounds play a crucial role.^[3–7] In these materials, the coexisted itinerant electrons and local magnetic moments (from localized f electrons) interact via the antiferromagnetic exchange coupling, resulting in the famous Kondo effect.^[8] Meanwhile, the Ruderman–Kittel–Kasuya–Yosida (RKKY) interaction, namely the exchange coupling among the local moments mediated by the itinerant electrons, competes with the Kondo effect.^[9] This competition gives rise to a rich phase diagram with an antiferromagnetic (AFM) quantum critical point (QCP) and various emergent phases nearby.^[3,10]

In the HF metals, experiments^[11,12] have provided strong evidence for local quantum criticality,^[13,14] which is characterized by the beyond-Landau physics of Kondo destruction at the AFM QCP. Across this local QCP, the Fermi surface jumps from large in the paramagnetic HF liquid phase to small in the AFM phase with Kondo destruction. A natural question arises: how does this local QCP connect to the conventional spin density wave (SDW) QCP described by the Hertz–Millis theory?^[15,16] A proposed global phase diagram^[17–20] makes this connection via the tuning of the quantum fluctuations in the local-moment magnetism. Besides the HF metals, it is also interesting to know whether a similar global phase diagram can be realized in Kondo insulators (KIs), where the chemical potential is inside the Kondo hybridization gap when the electron filling is commensurate. The KIs are band insulators where the band gap originates from nontrivial strong electron-correlation effects. A Kondo-destruction transition is expected to accompany the closure of the band gap. The question that remains open is: do the local moments immediately order or do they form a different type of magnetic state, such as spin liquid or valence bond solid (VBS), when the Kondo-destruction takes place?

Recent years have seen extensive studies about the effect of a fine spin–orbit coupling (SOC) on the electronic bands. In topological insulators (TIs), the bulk band gap opens due to a non-zero SOC, and there exist gapless surface states. The nontrivial topology of the band structure is protected by the time reversal symmetry (TRS). Even for a system with broken TRS, the conservation of combination of TRS and translational symmetry can give rise to a topological antiferromagnetic insulator (T-AFMI).^[21] In general, these TIs and TAFIs can be tuned to topologically trivial insulators via topological quantum phase transitions. But how the strong electron correlations influence the properties of these symmetry dictated topological phases and the related phase transitions is still under active discussion.

The SOC also has important effects in HF materials.^[20] For example, the SOC can produce a topologically nontrivial band structure and induce exotic Kondo physics.^[22,23] It may give rise to a topological Kondo insulator (TKI),^[24] which has been invoked to understand the resistivity plateau of the heavy-fermion SmB₆ at low temperatures.^[25]

From a more general perspective, the SOC provides an additional tuning parameter enriching the global phase diagram of the HF systems.^[20,26] Whether and how the topological nontrivial quantum phases can emerge in this phase diagram is a timely issue. Recent studies have advanced a Weyl–Kondo semimetal phase.^[27] The new heavy fermion compound Ce₃Bi₄Pd₃ displays thermodynamic^[28] and zero-field Hall transport^[29] properties that provide evidence for the salient features of the Weyl–Kondo semimetal. These measurements probe the linearly dispersing electronic excitations with a velocity that is renormalized by several orders of magnitude and singularities in the Berry-curvature distribution.

Theoretical studies are also of interest for a Kondo lattice model defined on a honeycomb lattice,^[30] which readily accommodates the SOC.^[31] In the diluted carrier limit, this model supports a nontrivial Dirac–Kondo semimetal (DKSM) phase, which can be tuned to a TKI by increasing the SOC.^[32] In Ref. [30], it was shown that, at half-filling, increasing the Kondo coupling induces a direct transition from a TI to a KI. A related model, with the conduction-electron part of the Hamiltonian described by a Haldane model^[33] on the honeycomb lattice, was subsequently studied.^[34]

Here we investigate the global phase diagram of a spin–orbit-coupled Kondo lattice model on the honeycomb lattice at half-filling. We show that the competing interactions in this model give rise to a very rich phase diagram containing a TI, a KI, and two AFM phases. We focus on discussing the influence of magnetic frustration on the phase diagram. In the TI, the local moments develop a VBS order. In the two AFM phases, the moments are ordered, respectively, in the plane of the honeycomb lattice (denoted as AFM_xy) and perpendicular to the plane (AFM_z). Particularly in the AFM_z phase, the conduction electrons may have a topologically nontrivial band structure, although the TRS is explicitly broken. This T-AFM_z state connects to the trivial AFM_z phase via a topological phase transition as the SOC is reduced.

The remainder of the paper is organized as follows. We start by introducing the model and our theoretical procedure in Section 2. In Section 3, we discuss the magnetic phase diagram of the Heisenberg model for the local moments. Next, we obtain the global phase diagram of the full model in Section 4. In Section 5, we examine the nature of the conduction-electron band structures in the AFM states, with a focus on their topological characters. We discuss the implications of our results in Section 6.

The model that we consider here is defined on an effective double-layer honeycomb lattice. The top layer contains conduction electrons realizing the Kane–Mele Hamiltonian.^[31] The conduction electrons are Kondo coupled (i.e., via AF exchange coupling J_K) to the localized magnetic moments in the bottom layer. The local moments interact among themselves through direct exchange interaction, as well as the conduction electron mediated RKKY interaction; the interactions are described by a simple J₁–J₂ model. Both the conduction bands and the localized bands are half-filled. This Kondo-lattice Hamiltonian takes the following form on the honeycomb lattice:

We use the spinon representation for , i.e., rewriting along with the constraint , where is the spinon operator. The constraint is enforced by introducing the Lagrange multiplier term in the Hamiltonian. To study both the non-magnetic and magnetic phases, We decouple the Heisenberg Hamiltonian into two channels

Fig. 1.

	Figure Option ViewDownloadNew Window
	Fig. 1. (a), (b) Definition of nearest neighboring and next nearest neighboring valence bond mean fields Qij. Filled and empty circles denote the two sublattices A and B, respectively. Different bond directions are labeled by different colors. (c) First Brilluion zone corresponds to the two-sublattice unit cell.

To take into account the Kondo hybridization and the possible magnetic order on an equal footing, we follow the treatment of the Heisenberg interaction as outlined in Eq. (2) and decouple the Kondo interaction as follows:

Because of the complexity of the full Hamiltonian, we start by setting J_K = 0 and discuss the possible ground-state phases of the J₁–J₂ Heisenberg model for the local moments. By treating the problem at the saddle-point level in Eq. (2), we obtain the phase diagram in the x– plane shown in Fig. 2. Here, the x-dependence is studied in the same spirit as that for the Shastry–Sutherland lattice in Ref. [19]. In the parameter regime explored, an AFM ordered phase (labeled as “AFM” in the figure) and a valence bond solid (VBS) phase are stabilized. The AF order stabilized is the two-sublattice Néel order on the honeycomb lattice, and the VBS order refers to covering of dimer singlets with for one out of the three NN bonds (e.g., ) and for all the NNN bonds. This VBS state spontaneously breaks the C₃ rotational symmetry of the lattice. We thus define the order parameter for the VBS state to be .

Fig. 2.

	Figure Option ViewDownloadNew Window
	Fig. 2. Ground-state phase diagram of the J1–J2 Hamiltonian for the local moments in the x– plane. A NN VBS and an AFM state are stabilized in the parameter regime shown.

In Fig. 3, we plot the evolution of VBS and AF order parameters Q and M as a function of . A direct first-order transition (signaled by the mid-point of the jump of the order parameters) between these two phases is observed for . For the sake of understanding the global phase diagram of the full Kondo–Heisenberg model, we limit our discussion to , where only the NN VBS is relevant. A different decoupling scheme approach has been used to study this model,^[37] and the obtained results are consistent with ours in the parameter regime of overlap. To fix the parameter x, we compare our results with those about the J₁–J₂ model derived from previous numerical studies. DMRG studies^[38] found that the AFM state is stabilized for , and VBS exists for , while in between the nature of the ground states is still under debate. In this parameter regime, the DMRG calculations suggest a plaquette resonating valence bond (RVB) state,^[38] while other methods implicate possibly spin liquids.^[39] In light of these numerical results, we take x = 0.4 in our calculations. This leads to a direct transition from AFM to VBS at , close to the values of phase boundaries of these two phases determined by other numerical methods.

Fig. 3.

	Figure Option ViewDownloadNew Window
	Fig. 3. Evolution of (a) the VBS order parameter Q and (b) the AFM order parameter M as a function of for x = 0.3, 0.4, 0.5.

We now turn to the global phase diagram of the full model by turning on the Kondo coupling. For definiteness, we set J₁ = 1 and consider t = 1 and . As prescribed in the previous section, we take x = 0.4. Similar considerations for y require that its value allows for quantum fluctuations in the form of Kondo-singlet formation. This has guided us to take y = 0.7 (see below). The corresponding phase diagram as a function of J_K and the frustration parameter is shown in Fig. 4.

Fig. 4.

	Figure Option ViewDownloadNew Window
	Fig. 4. Global phase diagram at T = 0 from the saddle-point calculations with x = 0.4 and y = 0.7. The ground states include the valence-bond solid (VBS) and Kondo insulator (KI), as well as two antiferromagnetic orders, T-AFMz and AFMxy, as described in Section 5.

In our calculation, the phase boundaries are determined by sweeping J_K while along multiple horizontal cuts for several fixed J₂/J₁ values, as shown in Fig. 5. For small J_K and large J₂/J₁, the local moments and the conduction electrons are still effectively decoupled. The conduction electrons form a TI for finite SOC, and the local moments are in the VBS ground state as discussed in the previous section. When both J_K and J₂/J₁ are small, the ground state is AFM. Due to the Kondo coupling, finite magnetization is induced for the conduction electrons. This opens an SDW gap in the conduction band, and therefore the ground state of the system is an AFM insulator. The SOC couples the rotational symmetry in the spin space to the one in the real space. As a consequence, the ordered moments in the AFM phase can be either along the z direction (AFM_z) or in the x–y plane (AFM_xy). For finite SOC, these two AFM states with different energies, which can be tuned by J_K. As shown in the phase diagram, the AFM phase contains two ordered states, the AFM_z and AFM_xy. They are separated by a spin reorientation transition at . For the value of SOC taken, the AFM state is topologically nontrivial, and is hence denoted as T-AFM_z state. The nature of this state and the associated topological phase transition is discussed in detail in the next section.

Fig. 5.

	Figure Option ViewDownloadNew Window
	Fig. 5. Evolution of parameters (a) b, (b) Q, (c) Mx, and (d) Mz as a function of JK for different ratios of J2/J1.

For sufficiently large J_K, the Kondo hybridization b is nonzero (see Fig. 5(a)), and the ground state is a KI. Note that for finite SOC, this KI does not have a topological nontrivial edge state, as a consequence of the topological no-go theorem.^[30,40,41] In our calculation at the saddle-point level, the KI exists for ; this provides the basis for taking y = 0.7, as noted earlier. Going beyond the saddle-point level, the dynamical effects of the Kondo coupling will appear, and we will expect the KI phase to arise for other choices of y.

Several remarks are in order. The phase diagram, Fig. 4, has a similar profile of the global phase diagram for the Kondo insulating systems.^[26,42] However, the presence of SOC has enriched the phase diagram. In the AF state, the ordered moment may lie either within the plane or be perpendicular to it. These two states have very different topological properties. We now turn to a detailed discussion of this last point.

In this section, we discuss the properties of the AFM_xy and AFM_z states, in particular to address their topological nature. For a clear discussion, we fix t = 1, J₁ = 1, and J₂ = 0. Since the Kondo hybridization is not essential to the nature of the AFM states, in this section we simply the discussion by setting y = 0.

We start by defining the order parameters of the two states

Fig. 6.

	Figure Option ViewDownloadNew Window
	Fig. 6. The conduction electron magnetization for the AFMxy and AFMz states at .

We then discuss the energy competition between the AFM_xy and AFM_z states. The conduction electron part of the mean-field Hamiltonian reads

The eigenenergies of the spinon band can be obtained in a similar way

By comparing the expressions of in Eqs. (11) and (12), we find that adding a small M_x is to increase the size of the gap at both of the two (inequivalent) Dirac points, thereby pushing the states further away from the Fermi-energy. While adding a small M_z is to enlarge the gap at one Dirac point but reduce the gap size at the other one. Therefore, an AFM_xy state is more favorable than the AFM_z state in lowering the energy of the conduction electrons .

Meanwhile, from Eqs. (13)–(15), we see that the overall effect of adding a magnetization of the conduction band, , is to increase the total energy (the main energy increase comes from the term). Because from the self consistent solution, as shown in Fig. 6, the energy increase of the AFM_z state is smaller than that in the AFM_xy state.

With increasing J_K, the two effects from the magnetic orders compete, resulting in different magnetic ground states as shown in Fig. 4. This analysis is further supported by our self-consistent mean-field calculation. In Fig. 7, we plot the energy difference between these two states as a function of J_K at several values. In the absence of SOC, the model has the spin SU(2) symmetry, and the AFM_z and AFM_xy states are degenerate with . For finite , at small J_K values, the energy gain from the term dominates, , and the ground state is an AFM_z state. With increasing J_K, the contribution from the term is more important. crosses zero to be negative, and the AFM_xy state is eventually energetically favorable for large J_K.

Fig. 7.

	Figure Option ViewDownloadNew Window
	Fig. 7. Energy difference between AFMz and AFMxy states as a function of JK for various values of spin–orbital coupling .

Next we discuss the topological nature of the AFM_z and AFM_xy states. In the absence of Kondo coupling J_K, the conduction electrons form a TI, which is protected by the TRS. There, the left- and right-moving edge states connecting the conduction and valence bands are respectively coupled to up and down spin flavors (eigenstates of the S^z operator) as the consequence of SOC, and these two spin polarized edge states do not mix.

Once the TRS is broken by the AFM order, generically, topologically nontrivial edge states are no longer guaranteed. However, in the AFM_z state, the structure of the Hamiltonian for the conduction electrons is the same as that in a TI. This is clearly shown in Eq. (10) the effect of magnetic order is only to shift to . In particular, the spin-up and spin-down sectors still do not mix with each other. Therefore, the two spin polarized edge states are still well defined as in the TI, and the system is topologically nontrivial, though without the protection of TRS. Note that the above analysis is based on assuming , where the bulk gap between the conduction and valence bands is finite. For , the bulk gap closes at one of the inequivalent Dirac points and the system is driven to a topologically trivial phase via a topological phase transition.^[30] We also note that a similar AFM_z state arises in a Kondo lattice model without SOC but with a Haldane coupling, as analyzed in Ref. [34].

For the AFM_xy state, we can examine the Hamiltonian for the conduction electrons in a similar way. As shown in Eq. (9), the transverse magnetic order M_x mixes the spin-up and spin-down sectors. As a result, a finite hybridization gap opens between the two edge states, making the system topologically trivial.

To support this analysis, we perform calculations of the energy spectra of the conduction electrons in the AFM_z and AFM_xy states, as shown in Eqs. (9) and (10), on a finite slab of size , with and . The boundary condition is chosen to be periodic along the x direction and open and zig-zag-type along the y direction. In Fig. 8, we show the energy spectra with three different sets of parameters: (a) , J_K = 0.4, , (b) , J_K = 0.4, , and (c) , J_K = 0.8, , which respectively correspond to the topologically trivial AFM_z state, topological AFM_z insulator, and AFM_xy state. As clearly seen, the gapless edge states only exist for parameter set (b), where the system is in the topological AFM_z state. Note that in this state, the spectrum is asymmetric with respect to the Brilluion zone boundary ( ), reflecting the explicit breaking of TRS. Based on our analysis and numerical calculations, we construct a phase diagram (as shown in Fig. 9) to illustrate the competition of these AFM states. As expected, the AFM_z state is stabilized for , and is topological for (above the red line).

Fig. 8.

	Figure Option ViewDownloadNew Window
	Fig. 8. Energy spectra of (a) the trivial AFMz state, (b) the topological AFMz insulator, and (c) the AFMxy state from finite slab calculations. Black lines denote the bulk states and red lines denote the edge states. The topological AFMz state is characterized by the gapless edge states. See text for detailed information on the parameters.

Fig. 9.

	Figure Option ViewDownloadNew Window
	Fig. 9. Phase diagram in the –JK plane showing the competition of various AFM states. The red line denotes a topological phase transition between the topological trivial and topological nontrivial AFMz states, and the black curve gives the boundary between the AFMz and AFMxy states. These two states become equivalent in the limit of .

We have discussed the properties of various phases in the ground-state phase diagram of the spin–orbit-coupled Kondo lattice model on the honeycomb lattice at half filling. We have shown how the competition of SOC, Kondo interaction, and magnetic frustration stabilizes these phases. For example, in the AFM phase the moments can order either along the z-direction or within the x–y plane. In our model, the AFM order is driven by the RKKY interaction, and the competition of SOC and Kondo interaction dictates the direction of the ordered magnetic moments.

Throughout this work, we have discussed the phase diagram of the model at half filling. The phase diagram away from half-filling is also an interesting problem. We expect that the competition between the AFM_z and AFM_xy states persist at generic fillings, but the topological feature will not. Another interesting filling would be the dilute-carrier limit, where a DKSM exists, and can be tuned to a TKI by increasing the SOC.^[32]

In this work, we have considered a particular type of SOC, which is inherent in the band structure of the itinerant electrons. In real materials, there are also SOC terms that involve the magnetic ions. Such couplings will lead to models beyond the current work, and may further enrich the global phase diagram.

Although the model in this work is defined on the honeycomb lattice, our conclusion on the global phase diagram is quite general, and will be important in understanding the nature of the transition between the Kondo insulating phase and the antiferromagnetic phase in real materials. For example, the Kondo insulator compound SmB₆ undergoes a magnetic transition under pressure.^[44] The Kondo-insulator-to-antiferromagnet transition may also be realized by doping CeNiSn with Pt or Pd ions since both CePtSn and CePdSn are antiferromagnetic at low temperatures.^[44–46] Nontrivial topological properties in the antiferromagnetic phase are expected given that the 5d electrons of CePtSn may contain a large SOC. Moreover, because the Kane–Mele model describes the electron states in graphene, our model may also shed light on the properties of graphene with 5d adatoms.^[47]

We have investigated the ground state phase diagram of a spin–orbit coupled Kondo lattice model at half-filling. The combination of SOC, Kondo and RKKY interactions produces various quantum phases, including a Kondo insulator, a topological insulator with VBS spin correlations, and two AFM phases. Depending on the strength of SOC, the magnetic moments in the AFM phase can be either ordered perpendicular to or in the x–y plane. We further show that the z-AFM state is topologically nontrivial for strong and moderate SOC, and can be tuned to a topologically trivial one via a topological phase transition by varying either the SOC or the Kondo coupling. Our results shed new light on the global phase diagram of heavy fermion materials.